Pose Compensation for Bimodal Speech Recognition

Lip reading has been proven to improve speech recognition accuracy in adverse environments. Most existing lip reading systems have frontal pose assumption, which makes it very difficult to use in tasks such as video transcription (speech recognition of the audio stream for video indexing and retrieval). In this paper, we propose a new method to compensate the lip pose change by exploiting the general symmetry of human face. From the imaging geometry we show that a frontal lip can be recovered from only one profile view. The resulting pose compensation method has the following advantages: (1) it only requires one profile image; (2) it does not need any 3D model; (3) it does not need an accurate lip shape contour. Experimental results are given to show the effectiveness of our method.